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Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 

tESS SJsp^TaS tZ^SfmSZ** within the statutory minium of thirty (30) days wil, be considered timely. . 

Failure to reply within the set or extended period for reply will, by statute, cause the appl.cat.on to become ^ANDONED p5 us C. § 1 33). 

- Any repty reSd oy the Office later than three months after the mailing date of this communication, even ,f timely tiled, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

I )|EI Responsive to communication(s) filed on 7/6/2004 . 

2a)l3 This action is FINAL. 2b)D This action is non-final. 

3D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 
closed in accordance with the practice under Ex parte Quayle, 1 935 CD. 1 1 , 453 O.G. 213. 
Disposition of Claims 

4) S Claim(s) 1^43 is/are pending in the application. 
4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) H3 Claim(s) 1^43 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

1 0)IEl The drawing(s) filed on 13 August 2001 is/are: a)E accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 85(a). 

I I )□ The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (0- 

a)D All b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2.D Certified copies of the priority documents have been received in Application No. . 

3 □ Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 1 9(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 

Attachment(s) 

1) □ Notice of References Cited (PTO-892) 4) □ Interview Summ^y (PTW13) Paper No(s) . 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) □ Notice of Informal Patent Application (PTO-1 52) 

3) (3 Information Disclosure Statement(s) (PTO-1449) Paper No(s) 12/27/04 . 6) □ Other: 



U.S. Patenl and Trademark Office Part nf Paner No 06242005 

PTO-326 (Rev. 04-01) Office Action Summary Part oT raper NO - "o^uud 



Application/Control Number: 09/830,899 
Art Unit: 2161 



Page 2 



Continued Examination 

A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 
1 . 1 7(e), was filed in this application after final rejection. Since this application is eligible for continued 
examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1 . 1 7(e) has been timely paid, the 
finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicants 
submission filed on 7/6/2004 has been entered. 

Claim Rejections - 35 USC §102 

1. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis 
for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed in the United 
States before the invention by the applicant for patent or (2) a patent granted on an application for patent by another filed in the 
United States before the invention by the applicant for patent, except that an international application filed under the treaty 
defined in section 351(a) shall have the effects for purposes of this subsection of an application filed in the United States only if 
the international application designated the United States and was published under Article 21(2) of such treaty in the English 
language. 

2. Claims 1-43 are rejected under 35 U.S.C. 102(e) as being anticipated by Pub No US 
2001/0000962 issued to Rajan (hereafter Rajan '962). 

Regarding claims 1,17 and 33, Rajan '962 discloses: 

(a) at least one multimedia information input interface receiving said multimedia information [Fig 1 
and paragraph 0040] 

(b) a computer processor [Fig 1], coupled to said at least one multimedia information input interface 
receiving said multimedia information therefrom, 
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processing said multimedia information by performing object extraction processing to generate 
multimedia object descriptions from said multimedia information [scene description information, 
paragraph 0042] 

processing said generated multimedia object descriptions by object hierarchy processing to generate 
multimedia object hierarchy descriptions indicative of an organization of said object descriptions 
[paragraph 0043] 

wherein at least one description record including said multimedia object descriptions and said 
multimedia object hierarchy descriptions is generated for content embedded within said multimedia 
information [paragraph 0044] 

( c ) a data storage system, operatively coupled to aid processor for storing at least said at least one 

description record [Fig 1, 105, 176, 186, 160, 170] 

Regarding claims 2, 18 and 34, Rajan '962 discloses image object description [paragraph 0043] 
Regarding claims 3, 7, 19 and 23, Rajan £ 962 discloses image segmentation [composition layer, 

paragraph 0045] and feature extraction [pitch of a sound, paragraph 0045] 

Regarding claims 4, 20, 35 and 39, Rajan £ 962 discloses size [scale, paragraph 0044] 
Regarding claims 5, 21 and 36, Rajan discloses spatial characteristics [paragraph 0044] 
Regarding claims 6, 22, 37, 41 and 43, Rajan '962 discloses logical descriptions [paragraph 

0043] 

Regarding claims 8 and 24, Rajan '962 discloses encoding the image object hierarchy 
descriptions [paragraph 0042] 

Regarding claims 9, 25 and 38, Rajan '962 discloses video object hierarchy descriptions 
[paragraph 0040] 
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Regarding claims 10 and 26, Rajan '962 discloses temporal video segmentation [paragraph 
0044] region segmentation [positioning parameters, paragraph 0043], feature extraction [paragraph 
0045] 

Regarding claims 1 1 and 27, Rajan £ 962 discloses an event feature description of time 
[paragraph 0044] and an object feature description of shape size position motion and time [paragraphs 
0042-0045] 

Regarding claims 12, 28, 40 and 42, Rajan '962 discloses physical event hierarchy descriptions 
[paragraph 0044] and temporal descriptions [paragraph 0044] 

Regarding claims 13, 14, 29 and 30, Rajan '962 discloses logical event hierarchy organization 
[paragraph 00043], video objects [paragraph 0040] and temporal descriptions [paragraph 0040] 

Regarding claims 15 and 31, Rajan '962 discloses temporal video segmentation [paragraph 
0041] grouping of video events into regions [paragraph 0043] and feature extraction [paragraph 0045] 

Regarding claims 16 and 32, Rajan '962 discloses encoded description information [paragraph 

0042] 

Response to Arguments 

Applicants arguments filed 12/1 1/2003 have been fully considered but they are not persuasive. 
Applicant Argues: 

Applicant states in the first paragraph on page 14 "Rajan is thus directed to a different problem, 
i.e., composing and presenting multimedia video, from that of the present invention, i.e., techniques for 
describing multimedia information content to enable intelligent searching of multimedia content via, 
e.g., the Internet. See Specification, p. 1 lines 1-4, p. 9, lines 23-29." 
Examiner Responds: 
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Examiner is not persuaded. In response to applicants argument that the references fail to show 
certain features of applicant's invention, it is noted that the features upon which applicant relies (i.e., 
searching via the Internet and the references to the specification) are not recited in the rejected claim(s). 
Although the claims are interpreted in light of the specification, limitations from the specification are not 
read into the claims. See In re Van Geuns, 988 F.2d 1 181, 26 USPQ2d 1057 (Fed. Cir. 1993). 
Applicant Argues: 

Applicant states in the second paragraph on page 14 "Another essential difference is the 
definition of object. Rajan merely uses object to refer to portions of media content, for example, an 
image region. In the present invention, object refers not only to a portion of some media content but 
also, importantly, objects in the real world that are depicted in the multimedia content. In addition, an 
object in Rajan can only have content of one media type. However, the present invention is generic to 
consider also portions of multimedia documents including different media. 
Examiner Responds: 

Examiner is not persuaded. In response to applicants argument that the references fail to show 
certain features of applicant's invention, it is noted that the features upon which applicant relies (i.e., 
depiction of real-world objects and generic multimedia) are not recited in the rejected claim(s). 
Although the claims are interpreted in light of the specification, limitations from the specification are not 
read into the claims. See In re Van Geuns, 988 F.2d 1 181, 26 USPQ2d 1057 (Fed. Cir. 1993). 

Furthermore. MPEP § 2106 requires Office personnel to give claims their broadest reasonable 
interpretation in light of the supporting disclosure. In re Morris, 127 F.3d 1048, 1054-55, 44 USPQ2d 
1023, 1027-28 (Fed. Cir. 1997). Applicant fails to point to the specification for a clear and precise 
definition of "object" per the present invention. Examiner thus has no option but to consult a common 
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dictionary which defines "object" as is most pertinent to the present invention as "a person or thing to 
which action, thought, or feeling is directed." Rajan's specification discloses the following in paragraph 
31: 

A multimedia terminal includes a terminal manager, a composition engines, content decodes, and a 
presentation engine. The composition engine maintains and updates a scene graph of the current 
objects, including their relative position in a scene, and their characteristics, to provide a list of objects 
to be displayed or used by the presentation engine to retrieve the decoded object data that is stored in 
respective composition buffers of content decoders. 

Clearly, above list of objects which are displayed in relative positions in order to create a scene 
are within the above common dictionary definition of "objects." 
Applicant Argues: 

Applicant states in he paragraph joining pages 14 and 15 "Rajan does not disclose or suggest 
processing said multimedia information by performing object extraction processing to generate 
multimedia object descriptions." 
Examiner Responds: 

Examiner is not persuaded. Applicant is referred to the following disclosure by Rajan, i.e., 
paragraphs 42-46 of the specification: 

6. The objects can exist independently, or be joined with other objects in a scene in a grouping known 
as a "composition " Visual objects in a scene are given a position in two- or three-dimensional space, 
while audio objects can be placed in a sound space. 

8. BIFS commands can add or delete objects from a scene, for example, or change the visual or acoustic 
properties of objects. BIFS commands also define, update, and position the objects. For example, a 
visual property such as the color or size of an object can be changes, or the object can be animated. 

24. The presentation engine should read the objects to be presented from the list, retrieve the objects 
from content decoders, and render the objects into appropriate buffers (e.g., display and audio buffers). 

3 1 . A multimedia terminal includes a terminal manager, a composition engine, content decoders, and a 
presentation engine. The composition engine maintains and updates a scene graph of the current objects, 
including their relative position in a scene and their characteristics, to provide a list of objects to be 



1 Webster's New World College Dictionary 



Application/Control Number: 09/830,899 
Art Unit: 2161 



Page 7 



displayed or played to the presentation engine. The list of objects is used by the presentation engine to 
retrieve the decoded object data that is stored in respective composition buffers of content decoders. 

43. The scene description information describes the logical structure of a scene, and indicates how 
objects are grouped together. Specifically, an MPEG-4 scene follows a hierarchical structure, which can 
be represented as a directed acyclic (tree) graph, where each node or a group of nodes, of the graph, 
represents a media object. The tree structure is not necessarily static, since node attributes (e.g., 
positioning parameters) can be changed while nodes can be added, replaced, or removed. 

44. The scene description information can also indicate how objects are positioned in space and time. In 
the MPEG-4 model, objects have both spatial and temporal characteristics. Each object has a local 
coordinate system in which the object has a fixed spatial-temporal location and scale. Objects are 
positioned in a scene by specifying a coordinate transformation from the object's local coordinate system 
into a global coordinate system defined by one more parent scene description nodes in the tree. 

45. The scene description information can also indicate attribute value selection. Individual media 
objects and scene description nodes expose a set of parameters to a composition layer through which 
part of their behavior can be controlled. Examples include the pitch of a sound, the color for a synthetic 
object, activation or deactivation of enhancement information for scaleable coding, and So forth. 

46. The scene description information can also indicate other transforms on media objects. The scene 
description structure and node semantics are heavily influenced by VRML, including its event model. 
This provides MPEG-4 with an extensive set of scene construction operators, including graphics 
primitives that can be used to construct sophisticated scenes. 



Rajan discloses per the above that the MPEG-4 Systems standard includes scene description 
information which describes the hierarchical structure of a scene and indicates how objects are grouped 
together in order to represent a multimedia object. Furthermore, nodes of groups of nodes may - 
represent a media object. The scene description information indicates how objects are positioned in 
space. The above disclosed multimedia composition engine updates a scene graph of the current 
objects to provide a list of objects to be displayed. Examiner maintains that above disclosure of 
objects which are included in a list, can be positioned in a scene in two- or three-dimensional space, 
can be deleted from a scene, have visual properties which can be changed, can be arranged 
hierarchically, can be positioned in time and space inherently read on "processing said multimedia 
information by performing object extraction processing to generate multimedia object descriptions." 
Applicant Argues: 
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Applicant states in the fourth paragraph on page 15 "Likewise paragraph 43 is cited as allegedly 
disclosing the claimed feature of processing said generated multimedia object descriptions by object 
hierarchy processing to generate multimedia object hierarchy" 
Examiner Responds: 

Examiner is not persuaded. In response to applicant's argument that the references fail to show 
certain features of applicant's invention, it is noted that the features upon which applicant relies (i.e., 
generation of multimedia object hierarchy) is not recited in the rejected claim(s). Although the claims 
are interpreted in light of the specification, limitations from the specification are not read into the claims. 
See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). 

Examiner responds to the complete claim limitation which is "processing said generated 
multimedia object descriptions by object hierarchy processing to generate multimedia object hierarchy 
descriptions indicative of an organization of said object descriptions." Applicant is referred to the 
following disclosure by Rajan, i.e., paragraphs 43-45 of the specification. 

43. The scene description information describes the logical structure of a scene, and indicates how 
objects are grouped together. Specifically, an MPEG-4 scene follows a hierarchical structure, which can 
be represented as a directed acyclic (tree) graph, where each node or a group of nodes, of the graph, 
represents a media object. The tree structure is not necessarily static, since node attributes (e.g., 
positioning parameters) can be changed while nodes can be added, replaced, or removed. 

44. The scene description information can also indicate how objects are positioned in space and time. In 
the MPEG-4 model, objects have both spatial and temporal characteristics. Each object has a local 
coordinate system in which the object has a fixed spatial-temporal location and scale. Objects are 
positioned in a scene by specifying a coordinate transformation from the object's local coordinate system 
into a global coordinate system defined by one more parent scene description nodes in the tree. 

45. The scene description information can also indicate attribute value selection. Individual media 
objects and scene description nodes expose a set of parameters to a composition layer through which 
part of their behavior can be controlled. Examples include the pitch of a sound, the color for a synthetic 
object, activation or deactivation of enhancement information for scaleable coding, and so forth. 



Applicant Argues: 
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Applicant states in the first paragraph on page 16 "This hierarchy is not described with sufficient 
detail, and certainly is not enabling to one of ordinary skill in the art. For at least this reason, the 
claimed "processing said generated multimedia object descriptions by object hierarchy processing to 
generate multimedia object hierarchy' is not disclosed for purposes on anticipation." 
Examiner Responds: 

Examiner is not persuaded. Applicant is referred to the following disclosure by Rajan, i.e., 
paragraph 40 of the specification: 

The present invention relates to a method and apparatus for composing and presenting multimedia 
video programs using the MPEG-4 standard. 

Examiner maintains that one of ordinary skill in the art is very familiar with an industry standard such 
as MPEG-4 which is described in ISO/IEC 14496-1 (1999) and in other publications per paragraph 3 
of the specification of the present application. 
Applicant Argues: 

Applicant states third paragraph on page 16, Applicant directs the Examiner to: 
http://wwwxhiariglionc.org/mpe^ 
httE;//www 
Examiner Responds: 

The above constitutes new matter and will not be entered into the record. Furthermore, above 
disclosure is objected to because it contains an embedded hyperlink and/or other form of browser- 
executable code. Applicant is required to delete the embedded hyperlink and/or other form of browser- 
executable code. SeeMPEP § 608.01. 
Applicant Argues: 

Applicant states in fourth paragraph on page 17 "Applicant again cannot find any reference in 
the above-cited paragraph to the claimed 'image segmentation' and 'feature extraction. 5 Indeed, there is 
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none. Accordingly, for at least this additional reason, claims 3, 7, 19 and 23 are further patentable over 
Rajan. 

Examiner Responds: 

Examiner is not persuaded. Examiner maintains the following disclosure by Rajan reads on 

image segmentation and feature extraction: 

4. The MPEG-4 communication standard allows a user to interact with video and audio objects within 
a scene, whether they are from conventional sources, such as moving video, or from synthetic 
(computer generated) sources. The user can modify scenes by deleting, adding or repositioning objects 
or changing the characteristics of the objects, such as size, color, and shape, for example. 

6. The objects can exist independently, or be joined with other objects in a scene in a grouping known 
as a "composition." Visual objects in a scene are given a position in two- or three-dimensional space, 
while audio objects can be placed in a sound space. 

8. BIFS commands can add or delete objects from a scene, for example, or changed the visual or 
acoustic properties of objects. BIFS commands also define, update, and position the objects. For 
example, a visual property such as the color or size of an object can be changed, or the object can be 
animated. 

Conclusion 

THIS ACTION IS MADE FINAL, Applicant is reminded of the extension of time policy as 
set forth in 37 CFR 1 .136(a). This is a first action is made final because Applicant did not make any 
claim amendments and merely repeated arguments previously presented. 

A shortened statutory period for reply to this final action is set to expire THREE MONTHS 
from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the 
mailing date of this final action and the advisory action is not mailed until after the end of the THREE- 
MONTH shortened statutory period, then the shortened statutory period will expire on the date the 
advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from 
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the mailing date of the advisory action. In no event, however, will the statutory period for reply expire 
later than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the examiner 
should be directed to Etienne LeRoux whose telephone number is (703) 305-0620. The examiner can 
normally be reached on Monday - Friday from 8:00 AM to 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, 
Safet Metjahic, can be reached on (703) 308-1436. 

Any inquiry of a general nature or relating to the status of this application or proceeding should 
be directed to the receptionist whose telephone number is (703) 305-3900. 
Etienne LeRoux 

6/24/2005 n s> 
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